Virtual avatars and physical object interaction in augmented reality applications

ABSTRACT

A method, an electronic device and computer readable medium for augmented reality object interaction. The method includes identifying an object within an area and properties associated with the object. The method also includes transmitting, to a remote electronic device, object information associated with the object. The method further includes receiving, from the remote electronic device, information including a virtual object, representing a remote user. Additionally, the method includes displaying the virtual object within the area. In response to determining that the virtual object is within a threshold distance from the object, the method also includes modifying the virtual object to interact with the object based on the properties of the object.

CROSS-REFERENCE TO RELATED APPLICATION AND CLAIM OF PRIORITY

This application claims priority under 35 U.S.C. § 119(e) to U.S.Provisional Patent Application No. 63/192,915 filed on May 25, 2021. Theabove-identified provisional patent applications is hereby incorporatedby reference in its entirety.

TECHNICAL FIELD

This disclosure relates generally to imaging and visualization systems.More specifically, this disclosure relates to virtual avatars andphysical object interaction in augmented reality applications.

BACKGROUND

Augmented reality (AR) and virtual reality (VR) are emerging as new waysof experiencing immersive video due to the ready availability ofpowerful handheld devices such as smartphones. The AR and VR experiencesprovide immersive “real life,” “being there” experience for consumers.For example, during a AR or VR call a first user can interact with andview another user or an avatar that represents the other user.

SUMMARY

This disclosure provides virtual avatars and physical object interactionin augmented reality applications.

In a first embodiment, a method includes identifying an object within anarea and properties associated with the object. The method also includestransmitting, to a remote electronic device, object informationassociated with the object. The method further includes receiving, fromthe remote electronic device, information including a virtual object,representing a remote user. Additionally, the method includes displayingthe virtual object within the area. The method also includes that inresponse to determining that the virtual object is within a thresholddistance from the object, modifying the virtual object to interact withthe object based on the properties of the object.

In a second embodiment, an electronic device includes a communicationinterface and a processor. The processor is configured to identify anobject within an area and properties associated with the object. Theprocessor is also configured to transmit, to a remote electronic device,object information associated with the object. The processor is furtherconfigured to receive, from the remote electronic device, informationincluding a virtual object, representing a remote user. Additionally,the processor is configured to display the virtual object within thearea. In response to determining that the virtual object is within athreshold distance from the object, the processor is configured tomodify the virtual object to interact with the object based on theproperties of the object.

In a third embodiment, a non-transitory machine-readable medium containsinstructions that, when executed, cause at least one processor of anelectronic device to identify an object within an area and propertiesassociated with the object. The medium also contains instructions that,when executed, cause the at least one processor to transmit, to a remoteelectronic device, object information associated with the object. Themedium further contains instructions that, when executed, cause the atleast one processor to receive, from the remote electronic device,information including a virtual object, representing a remote user.Additionally, the medium contains instructions that, when executed,cause the at least one processor to display the virtual object withinthe area. In response to determining that the virtual object is within athreshold distance from the object, the medium contains instructionsthat, when executed, cause the at least one processor to modify thevirtual object to interact with the object based on the properties ofthe object.

Other technical features may be readily apparent to one skilled in theart from the following figures, descriptions, and claims.

Before undertaking the DETAILED DESCRIPTION below, it may beadvantageous to set forth definitions of certain words and phrases usedthroughout this patent document. The term “couple” and its derivativesrefer to any direct or indirect communication between two or moreelements, whether or not those elements are in physical contact with oneanother. The terms “transmit,” “receive,” and “communicate,” as well asderivatives thereof, encompass both direct and indirect communication.The terms “include” and “comprise,” as well as derivatives thereof, meaninclusion without limitation. The term “or” is inclusive, meaningand/or. The phrase “associated with,” as well as derivatives thereof,means to include, be included within, interconnect with, contain, becontained within, connect to or with, couple to or with, be communicablewith, cooperate with, interleave, juxtapose, be proximate to, be boundto or with, have, have a property of, have a relationship to or with, orthe like. The term “controller” means any device, system or part thereofthat controls at least one operation. Such a controller may beimplemented in hardware or a combination of hardware and software and/orfirmware. The functionality associated with any particular controllermay be centralized or distributed, whether locally or remotely. Thephrase “at least one of,” when used with a list of items, means thatdifferent combinations of one or more of the listed items may be used,and only one item in the list may be needed. For example, “at least oneof: A, B, and C” includes any of the following combinations: A, B, C, Aand B, A and C, B and C, and A and B and C.

Moreover, various functions described below can be implemented orsupported by one or more computer programs, each of which is formed fromcomputer readable program code and embodied in a computer readablemedium. The terms “application” and “program” refer to one or morecomputer programs, software components, sets of instructions,procedures, functions, objects, classes, instances, related data, or aportion thereof adapted for implementation in a suitable computerreadable program code. The phrase “computer readable program code”includes any type of computer code, including source code, object code,and executable code. The phrase “computer readable medium” includes anytype of medium capable of being accessed by a computer, such as readonly memory (ROM), random access memory (RAM), a hard disk drive, acompact disc (CD), a digital video disc (DVD), or any other type ofmemory. A “non-transitory” computer readable medium excludes wired,wireless, optical, or other communication links that transporttransitory electrical or other signals. A non-transitory computerreadable medium includes media where data can be permanently stored andmedia where data can be stored and later overwritten, such as arewritable optical disc or an erasable memory device.

Definitions for other certain words and phrases are provided throughoutthis patent document. Those of ordinary skill in the art shouldunderstand that in many if not most instances, such definitions apply toprior as well as future uses of such defined words and phrases.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present disclosure and itsadvantages, reference is now made to the following description taken inconjunction with the accompanying drawings, in which like referencenumerals represent like parts:

FIG. 1 illustrates an example communication system in accordance with anembodiment of this disclosure;

FIG. 2 illustrates an example network configuration including electronicdevices in accordance with this disclosure;

FIG. 3 illustrates a block diagram of a communication system for objectinteraction between a virtual object and a physical object in anaugmented reality environment in accordance with this disclosure;

FIG. 4 illustrates example properties associated with a physical objectin accordance with this disclosure;

FIG. 5 illustrates an example block diagram for adjusting a pose of anavatar in accordance with this disclosure;

FIGS. 6A-7G illustrate example images of object interaction between avirtual avatar and a physical object in accordance with this disclosure;and

FIG. 8 illustrates an example method for augmented reality objectinteraction in accordance with this disclosure.

DETAILED DESCRIPTION

FIGS. 1 through 8 , discussed below, and the various embodiments used todescribe the principles of the present disclosure in this patentdocument are by way of illustration only and should not be construed inany way to limit the scope of the disclosure. Those skilled in the artwill understand that the principles of the present disclosure may beimplemented in any suitably-arranged system or device.

An electronic device, according to embodiments of the presentdisclosure, can include a personal computer (such as a laptop, adesktop), a workstation, a server, a television, an appliance, and thelike. In certain embodiments, an electronic device can be a portableelectronic device such as a portable communication device (such as asmartphone or mobile phone), a laptop, a tablet, an electronic bookreader (such as an e-reader), a personal digital assistants (PDAs), aportable multimedia player (PMP), an MP3 player, a mobile medicaldevice, a virtual reality/augmented reality headset (such as a headmounted display, glasses, and the like), a portable game console, acamera, and a wearable device, among others. Additionally, theelectronic device can be at least one of a part of a piece of furnitureor building/structure, an electronic board, an electronic signaturereceiving device, a projector, or a measurement device. The electronicdevice is one or a combination of the above-listed devices.Additionally, the electronic device as disclosed herein is not limitedto the above-listed devices and can include new electronic devicesdepending on the development of technology. It is noted that as usedherein, the term “user” may denote a human or another device (such as anartificial intelligent electronic device) using the electronic device.

Virtual reality (VR) is a fully rendered version of a visual scene,where the entire scene is computer generated. Augmented reality (AR) isan interactive experience of a real-world environment where objects thatreside in the real-world environment are augmented with virtual objects,virtual information, or both. In certain embodiments, AR and VR includeboth visual and audio experiences. A visual rendering is designed tomimic the visual stimuli, and if available audio sensory stimuli, of thereal world as naturally as possible to an observer or user as the usermoves within the limits defined by the application or the AR or VRscene. For example, VR places a user into immersive worlds that respondto detected head movements of a user. At the video level, VR is achievedby providing a video experience that covers as much of the field of view(FOV) as possible together with the synchronization of the viewing angleof the rendered video with the head movements. In an AR environment, auser both perceives the real world and is provided with additionalcomputer generated information that enhances or modifies a user'sperception of the real world.

Many different types of devices are able to provide the immersiveexperience associated with AR or VR. One example device is ahead-mounted display (HMD). An HMD represents one of many types ofdevices that provide AR and VR experiences to a user. An HMD is a devicethat enables a user to view the VR scene and adjust the displayedcontent based on movements of the head of the user. Typically, an HMDrelies either on a dedicated screen that is integrated into a device andconnected with an external computer (tethered) or on a device, such as asmartphone, that is inserted into the HMD (untethered). The firstapproach utilizes one or more lightweight screens and benefits from ahigh computing capacity. In contrast, the smartphone-based systemsutilize higher mobility and can be less expensive to produce. In bothinstances, the video experience generated is the same.

Another example of a device that provide the immersive experienceassociated with AR is AR glasses (also referred to as smart glasses,wearable computer glasses, and the like). AR glasses include transparentdisplay screen (such as a transparent heads up display) that providesboth a view of the physical world and the ability to display virtualobjects or information for the user to view. Certain AR glasses includesensors and processors that modify the virtual objects based onmovements of the user and the objects of the real world.

For yet another example of a device that provide the immersiveexperience associated with AR is an electronic device with both a cameraand a display. The camera can capture images of the real world, such asthe environment around the electronic device. The captured images arethen displayed on the display along with one or more added virtualobjects or information for the user to view. As discussed above, variousother electronic devices can provide both VR and AR environments for auser.

Multi-user applications (such as communications, games, wordproductivity, and the like), can provide platforms for two or more usersto interact in an AR environment. One of the users is denoted as theuser (or viewer) and located at one location while the other user(s) aredenoted as remote user(s) and located at one or more differentlocations. An AR environment enables the viewer, via an electronicdevice, to both view their environment and a virtual object. Certainthree dimensional (3D) spatial rendering systems and applications enablea remote user to control a virtual object in the space of another user.In certain embodiments, the remote object is a virtual avatar, which isa graphical representation of the remote user. For example, an avatar,representing a remote user can be displayed in the AR environment of theviewer. That is, the viewer in an AR environment, can view a virtualobject (such as an avatar) that is controlled by a remote user withintheir real environment. The remote user can control the movement,position, gestures, and the like of their avatar in an AR environmentfor another user to view and interact with. However, the remote user maybe unaware of the physical objects at the location of the viewer. Assuch, while the remote user controls the avatar, the avatar could bepositioned at least partially in a wall (such as illustrated in FIG. 6C)or within furniture (such as illustrated in FIG. 7D) that is located atthe viewers location. As such, the avatar can intersect and overlap realworld objects in the viewers location, causing a break in the augmentedreality experience as well as can provide improper communication betweenthe remote user, that is represented as the avatar, and the viewer.

Certain embodiments of the present disclosure provide for a situation inwhich a remote user, while controlling a virtual object at the locationof the viewer in an AR environment, is unaware of the location of thephysical objects at the viewers location and their correspondingproperties. Certain embodiments of the present disclosure also or in thealternative provide for a situation, when a virtual object that iscontrolled by a remote user, intersects a real world object (alsoreferred to as a physical object), the virtual object impairs the ARenvironment of the viewer.

Therefore, embodiments of the present disclosure provide systems andmethods for recognizing real world objects at the location of theviewer. Based on the recognized real world objects and propertiesassociated with the objects, certain embodiments enable the virtualobject to meaningfully interact with the real-world object. Embodimentsof the present disclosure also provide to the remote user theapproximate location of a real world object relative to the virtualobject and the information associated with the real world objects, forenabling the remote user to interact with the real world object. Theremote user can then move the avatar to a new location or alter the poseof their avatar. Embodiments of the present disclosure further providesystems and methods that modify the pose or location of the avatar withor without receiving a direction from the remote user for interactingwith the real world objects. The modifications can include altering thepose of the virtual object, adding shadows around the real world objectbased on the location of the virtual object, adding sound when virtualobject interacts with the real world object, and the like.

FIG. 1 illustrates an example communication system 100 in accordancewith an embodiment of this disclosure. The embodiment of thecommunication system 100 shown in FIG. 1 is for illustration only. Otherembodiments of the communication system 100 can be used withoutdeparting from the scope of this disclosure.

The communication system 100 includes a network 102 that facilitatescommunication between various components in the communication system100. For example, the network 102 can communicate Internet Protocol (IP)packets, frame relay frames, Asynchronous Transfer Mode (ATM) cells, orother information between network addresses. The network 102 includesone or more local area networks (LANs), metropolitan area networks(MANs), wide area networks (WANs), all or a portion of a global networksuch as the Internet, or any other communication system or systems atone or more locations.

In this example, the network 102 facilitates communications between aserver 104 and various client devices 106-116. The client devices106-116 may be, for example, a smartphone, a tablet computer, a laptop,a personal computer, a wearable device, a head-mounted display (HMD),virtual assistant, or the like. The server 104 can represent one or moreservers. Each server 104 includes any suitable computing or processingdevice that can provide computing services for one or more clientdevices, such as the client devices 106-116. Each server 104 could, forexample, include one or more processing devices, one or more memoriesstoring instructions and data, and one or more network interfacesfacilitating communication over the network 102.

Each client device 106-116 represents any suitable computing orprocessing device that interacts with at least one server (such as theserver 104) or other computing device(s) over the network 102. In thisexample, the client devices 106-116 include a desktop computer 106, amobile telephone or mobile device 108 (such as a smartphone), a PDA 110,a laptop computer 112, a tablet computer 114, and an HMD 116. However,any other or additional client devices could be used in thecommunication system 100. Smartphones represent a class of mobiledevices 108 that are handheld devices with mobile operating systems andintegrated mobile broadband cellular network connections for voice,short message service (SMS), and Internet data communications. The HMD116 can display an AR environment or a VR environment.

In this example, some client devices 108-116 communicate indirectly withthe network 102. For example, the client devices 108 and 110 (mobiledevice 108 and PDA 110, respectively) communicate via one or more basestations 117, such as cellular base stations or eNodeBs (eNBs). Also,the laptop computer 112, the tablet computer 114, and the HMD 116communicate via one or more wireless access points 118, such as IEEE802.11 wireless access points. Note that these are for illustration onlyand that each client device 106-116 could communicate directly with thenetwork 102 or indirectly with the network 102 via any suitableintermediate device(s) or network(s).

In some embodiments, any of the client devices 106-116 transmitsinformation securely and efficiently to another device, such as, forexample, the server 104. Also, any of the client devices 106-116 cantrigger the information transmission between itself and server 104. Anyof the client devices 106-114 can function as an AR or VR display whenattached to a headset via brackets, and function similar to HMD 116. Forexample, the mobile device 108 when attached to a bracket system andworn over the eyes of a user can function similarly as the HMD 116. Themobile device 108 (or any other client device 106-116) can trigger theinformation transmission between itself and the server 104.

Although FIG. 1 illustrates one example of a communication system 100,various changes can be made to FIG. 1 . For example, the communicationsystem 100 could include any number of each component in any suitablearrangement. In general, computing and communication systems come in awide variety of configurations, and FIG. 1 does not limit the scope ofthis disclosure to any particular configuration. While FIG. 1illustrates one operational environment in which various featuresdisclosed in this document can be used, these features could be used inany other suitable system.

FIG. 2 illustrates an example network configuration 200 includingelectronic devices in accordance with this disclosure. The embodiment ofthe network configuration 200 shown in FIG. 2 is for illustration only.Other embodiments of the network configuration 200 could be used withoutdeparting from the scope of this disclosure.

According to embodiments of this disclosure, an electronic device 201 isincluded in the network configuration 200. The electronic device 201 canbe similar to any of the client devices 106-116 of FIG. 1 . In certainembodiments, the electronic device 201 for rendering an AR environmentfor a viewer. The electronic device 201 can include at least one of abus 210, a processor 220, a memory 230, an input/output (I/O) interface250, a display 260, a communication interface 270, one or more sensors280, a speaker 290, and a camera 295. In some embodiments, theelectronic device 201 may exclude at least one of these components ormay add at least one other component. The bus 210 includes a circuit forconnecting the components 220-295 with one another and for transferringcommunications (such as control messages and/or data) between thecomponents.

The processor 220 includes one or more of a central processing unit(CPU), a graphics processor unit (GPU), an application processor (AP),or a communication processor (CP). The processor 220 is able to performcontrol on at least one of the other components of the electronic device201 and/or perform an operation or data processing relating tocommunication. In certain embodiments, the processor 220 identifies aphysical object in the real world and properties of the physical object,such as whether the object is hard or soft. In certain embodiments, theprocessor 220 modifies virtual objects that are rendered in the ARenvironment based on the properties associated with the identifiedobject.

The memory 230 can include a volatile and/or non-volatile memory. Forexample, the memory 230 can store commands or data related to at leastone other component of the electronic device 201. According toembodiments of this disclosure, the memory 230 can store software and/ora program 240. The program 240 includes, for example, a kernel 241,middleware 243, an application programming interface (API) 245, and/oran application program (or “application”) 247. At least a portion of thekernel 241, middleware 243, or API 245 may be denoted as an operatingsystem (OS).

The kernel 241 can control or manage system resources (such as the bus210, processor 220, or memory 230) used to perform operations orfunctions implemented in other programs (such as the middleware 243, API245, or application 247). The kernel 241 provides an interface thatallows the middleware 243, the API 245, or the application 247 to accessthe individual components of the electronic device 201 to control ormanage the system resources. The application 247 includes one or moreapplications for object detection and recognition as discussed below.These functions can be performed by a single application or by multipleapplications in which each carries out one or more of these functions.The middleware 243 can function as a relay to allow the API 245 or theapplication 247 to communicate data with the kernel 241, for instance. Aplurality of applications 247 can be provided. The middleware 243 isable to control work requests received from the applications 247, suchas by allocating the priority of using the system resources of theelectronic device 201 (like the bus 210, the processor 220, or thememory 230) to at least one of the plurality of applications 247. TheAPI 245 is an interface allowing the application 247 to controlfunctions provided from the kernel 241 or the middleware 243. Forexample, the API 245 includes at least one interface or function (suchas a command) for filing control, window control, image processing, ortext control.

The I/O interface 250 serves as an interface that can, for example,transfer commands or data input from a user or other external devices toother component(s) of the electronic device 201. The I/O interface 250can also output commands or data received from other component(s) of theelectronic device 201 to the user or the other external device.

The display 260 includes, for example, a liquid crystal display (LCD), alight emitting diode (LED) display, an organic light emitting diode(OLED) display, a quantum-dot light emitting diode (QLED) display, amicroelectromechanical systems (MEMS) display, or an electronic paperdisplay. The display 260 can also be a depth-aware display, such as amulti-focal display. The display 260 is able to display, for example,various contents (such as text, images, videos, icons, or symbols) tothe user. The display 260 can include a touchscreen and may receive, forexample, a touch, gesture, proximity, or hovering input using anelectronic pen or a body portion of the user. The display 260 can be atransparent display or a heads up display, to enable the viewer to bothview the real world and virtual objects at similar times.

The communication interface 270, for example, is able to set upcommunication between the electronic device 201 and an externalelectronic device (such as an electronic device 202, a second electronicdevice 204, or a server 206). For example, the communication interface270 can be connected with a network 262 or 264 through wireless or wiredcommunication to communicate with the external electronic device. Thecommunication interface 270 can be a wired or wireless transceiver orany other component for transmitting and receiving signals, such asimages.

The wireless communication is able to use at least one of, for example,long term evolution (LTE), long term evolution-advanced (LTE-A), 5thgeneration wireless system (5G), millimeter-wave or 60 GHz wirelesscommunication, Wireless USB, code division multiple access (CDMA),wideband code division multiple access (WCDMA), universal mobiletelecommunication system (UMTS), wireless broadband (WiBro), or globalsystem for mobile communication (GSM), as a cellular communicationprotocol. The wired connection can include, for example, at least one ofa universal serial bus (USB), high definition multimedia interface(HDMI), recommended standard 232 (RS-232), or plain old telephoneservice (POTS). The network 262 or 264 includes at least onecommunication network, such as a computer network (like a local areanetwork (LAN) or wide area network (WAN)), Internet, or a telephonenetwork.

The electronic device 201 further includes one or more sensors 280 thatcan meter a physical quantity or detect an activation state of theelectronic device 201 and convert metered or detected information intoan electrical signal. For example, one or more sensors 280 can includeone or more buttons for touch input, a gesture sensor, a gyroscope orgyro sensor, an air pressure sensor, a magnetic sensor or magnetometer,an acceleration sensor or accelerometer, a grip sensor, a proximitysensor, a color sensor (such as a red green blue (RGB) sensor), abio-physical sensor, a temperature sensor, a humidity sensor, anillumination sensor, an ultraviolet (UV) sensor, an electromyography(EMG) sensor, an electroencephalogram (EEG) sensor, an electrocardiogram(ECG) sensor, an infrared (IR) sensor, an ultrasound sensor, an irissensor, or a fingerprint sensor. The sensor(s) 280 can further includean inertial measurement unit, which can include one or moreaccelerometers, gyroscopes, and other components. In addition, thesensor(s) 280 can include a control circuit for controlling at least oneof the sensors included here. Any of these sensor(s) 280 can be locatedwithin the electronic device 201.

The electronic device 201 further includes one or more speakers 290 thatconvert electrical signals into sound. The electronic device 201 canalso include a microphone similar to a dynamic microphone, a condensermicrophone, a piezoelectric microphone, or the like.

Additionally, the electronic device 201 includes one or more cameras295, or other imaging sensors, for capturing images or video of theenvironment. The camera 295 is able to view an environment andgenerating a sequence of images of video of the environment. The camera295 can include one or more of a color camera (such as an RGB camera), avideo camera, a depth camera, a motion sensor, radar, sonar, infrared(IR), and the like.

The external electronic device 202 and the external electronic device204 can be similar to any of the client devices 106-116 of FIG. 1 . Theserver 206 can be similar to the server 104 of FIG. 1 . According tocertain embodiments of this disclosure, the server 206 includes a groupof one or more servers. The external electronic devices 202 and 204 andthe server 206 each can be a device of the same or a different type fromthe electronic device 201.

According to certain embodiments of this disclosure, all or some of theoperations executed on the electronic device 201 can be executed onanother or multiple other electronic devices (such as the electronicdevices 202 and 204 or server 206). Further, according to certainembodiments of this disclosure, when the electronic device 201 shouldperform some function or service automatically or at a request, theelectronic device 201, instead of executing the function or service onits own or additionally, can request another device (such as electronicdevices 202 and 204 or server 206) to perform at least some functionsassociated therewith. The other electronic device (such as electronicdevices 202 and 204 or server 206) is able to execute the requestedfunctions or additional functions and transfer a result of the executionto the electronic device 201. The electronic device 201 can provide arequested function or service by processing the received result as it isor additionally. To that end, a cloud computing, distributed computing,or client-server computing technique may be used, for example. WhileFIG. 2 shows that the electronic device 201 includes the communicationinterface 270 to communicate with the external electronic device 204 orserver 206 via the network 262 or 264, the electronic device 201 may beindependently operated without a separate communication functionaccording to some embodiments of this disclosure.

The external electronic devices 202 and 204 can include the same orsimilar components 210-295 as the electronic device 201 (or a suitablesubset thereof). Additionally, the external electronic devices 202 and204 can be a wearable device or an electronic device-mountable wearabledevice (such as an HMD). When the electronic device 201 is mounted inthe electronic device 202 (such as the HMD), the electronic device 201can communicate with the electronic device 202 through the communicationinterface 270. The electronic device 201 can be directly connected withthe electronic device 202 to communicate with the electronic device 202without involving with a separate network. In certain embodiments, theexternal electronic devices 202 and 204 can be used by a remote user tocontrol a virtual object that is displayed on the electronic device 201.

The server 206 can support to drive the electronic device 201 byperforming at least one of operations (or functions) implemented on theelectronic device 201. For example, the server 206 can include aprocessing module or processor that may support the processor 220implemented in the electronic device 201. In certain embodiments, theserver 206 performs the object recognition based on images or videocaptured by the camera 295.

Although FIG. 2 illustrates one example of a network configuration 200including an electronic device 201, various changes may be made to FIG.2 . For example, the network configuration 200 could include any numberof each component in any suitable arrangement. In general, computing andcommunication systems come in a wide variety of configurations, and FIG.2 does not limit the scope of this disclosure to any particularconfiguration. Also, while FIG. 2 illustrates one operationalenvironment in which various features disclosed in this document can beused, these features could be used in any other suitable system.

FIG. 3 illustrates a block diagram of a communication system 300 forobject interaction between a virtual object and a physical object in anAR environment in accordance with this disclosure. The communicationsystem 300 may be implemented in one of the client devices 106-116 orthe server 104 of FIG. 1 , the electronic device 201 or the server 206of FIG. 2 , or any combination thereof.

The communication system 300 includes an electronic device 310 and aremote electronic device 340. The electronic device 310 can be the sameas or similar to the electronic device 201 while the remote electronicdevice 340 can be similar to the electronic device 202 or 204 of FIG. 2. The embodiment of the communication system 300 shown in FIG. 3 is forillustration only. Other embodiments can be used without departing fromthe scope of the present disclosure.

In certain embodiments, the electronic device 310 and the remoteelectronic device 340 are in communication such as via a network 102 ofFIG. 1 . For example, the electronic device 310 and the remoteelectronic device 340 can be using a video conference application suchthat the electronic device 310 displays a virtual representation of aremote user. In this example, the viewer of the electronic device 310can view in an AR environment both their physical environment and withthe addition of a virtual object, such as an avatar, representing theuser of the remote electronic device 340.

The electronic device 310 includes an information repository 320. Theinformation repository 320 includes object information 322 and physicalproperty information 324. The object information 322 is used for objectrecognition. For example, the electronic device 310 can identify acategory or type of a physical object based on the object information322. The physical property information 324 includes various propertiesassociated with an object such as whether the object is hard, soft,dense, possess an ability to absorb a force, elasticity, and the like.

In certain embodiments, a user can manually augment the informationrepository 320. For example, the user can add objects to the objectinformation 322 and physical property information 324 based on thephysical objects at their location. For example, the user can capture animage of various physical objects and provide a description of theobject and various physical properties associated with the object. Incertain embodiments, the user can also modify existing objects andproperties within the information repository 320.

In certain embodiments, the properties can be accessed by other userswho have similar objects. For example, the information repository can bemaintained in a remote server and accessed by various devices for objectrecognition. In certain embodiments, the information repository 320 alsostores collider information.

The electronic device 310 obtains camera data 312. The camera data 312can be captured from the camera 295 of FIG. 2 . The camera data 312includes a video or images of the physical location (environment) aroundthe electronic device 310. The camera data can include physical objectssuch as walls, furniture, and other real world physical objects.

As illustrated, the object recognition engine 314 is performed on theelectronic device 310. In certain embodiments, the object recognitionengine 314 is performed remotely such as on a server 104.

In certain embodiments, the object recognition engine 314 uses machinelearning to identify an object. The object recognition engine 314 canalso use computer vision when identifying an object. The objectrecognition engine 314 can identify the object type and volume (size) ofthe object. In certain embodiments, the object recognition engine 314recognizes the real world objects in their space using data from theinformation repository 320. For example, if the camera data 312 includesan object, such as a sofa, the object recognition engine 314 canidentify that the object is a sofa based on object information 322. Incertain embodiments, the object recognition engine 314 compares thedetected object to various objects in the object information 322 inorder to identify or categorize the object.

In certain embodiments, the object recognition engine 314 alsoidentifies properties associated with an object such as whether theobject is hard, soft, dense, ability to absorb a force, elasticity andthe like. The virtual object could interact with an object according tothe identified properties to increase the immersion of the ARenvironment. For example, a virtual object, such as an avatar, sittingon a hard wooden chair would appear differently than a virtual objectsitting on a plush sofa.

After determining that the physical object is a sofa, the objectrecognition engine 314 parses the physical properties information 324 toidentify various physical properties associated with a sofa. Forexample, the object recognition engine 314 can determine that the sofais soft and deformable.

The electronic device 310 can generate collision primitives around anidentified object. The collision primitives provide an indication to theviewer, or to the remote user, regarding the size of the physicalobject. In certain embodiments, a partial collision primitive isgenerated when the object is partially covered by another object. Thecollision primitives can be a shape such as planes, prisms or spheresthat is formed around a physical object. FIGS. 6B and 7B illustrateexample collision primitives.

After identifying the object and its properties, the object transformengine 316 identifies the relative position of the object relative tothe viewer and the virtual object representing the remote user. Theobject transform engine 316 can also identify an approximate volume ofthe object.

The information 330 associated with the object is then transmitted tothe remote electronic device 340. The information 330 can include thelocation of the object, the size of the object, a type of object, andproperties of the object. The information 330 can also include thecollision primitives. In certain embodiments, the information 330includes spatial volumetric information about the object, to enable theremote user knowledge about the physical environment in proximity of theviewer, the virtual object, or both.

The remote electronic device 340 receives the information 330 from theelectronic device 310. The avatar transform engine 342 controls theposition of the virtual object relative to the physical objects or theviewer. For example, if there is a spatial reference point, such as ananchor point, the virtual object can be initially positioned at alocation relative to the anchor point. In this example, the virtualobject is initially positioned at a location relative to a physicalobject at the location of the viewer.

The object representation engine 344 provides an indication to theremote user as to the location of the one or more physical objects. Incertain embodiments, if the remote electronic device 340 includes adisplay (similar to the display 260 of FIG. 2 ), the objectrepresentation engine 344 displays the physical object at a locationrelative to the virtual object. For example, the object representationengine 344 generates and renders, on a display, some representation ofthe physical object at a position that is relative to the virtual objectas perceived by the viewer. By displaying a representation of thephysical object to the remote user, the object representation engine 344enables the remote user to have a spatial understanding of the physicalobjects at the viewers location. FIG. 7C illustrates a view of theremote user with a virtual object representing the physical object atthe viewers location, while FIG. 7D illustrates a view of what theviewer observes at of a virtual object that is overlapping a physicalobject at the viewers location.

Once the remote user knows of the positions of the physical objectsrelative to the position of the virtual object at the viewers location,the remote user can either manually control the virtual object 346 orgenerate an indication 348. The remote user can then manually controlthe virtual object 346 by moving the virtual object to a new location toavoid or interact with known physical objects. For example, the remoteuser can position the virtual object at a location in the viewerslocation that does not overlap with the physical objects of the viewer.In another example, the remote user can alter the position of thevirtual object. That is, if the virtual object is an avatar that ispositioned on or near a chair, the remote user can modify the avatarfrom a standing position to a seated position in the chair. Instructingthe avatar to sit in the chair provides an element of authenticity tothe augmented reality that the avatar is actually in the location of theviewer. The remote electronic device 340 transmits the instructions 334for manually controlling the virtual object to the electronic device310.

The remote user can select a displayed primitive, representing aphysical object at the viewers location. The remote electronic device340 can then display a notification. The notification specifies one ormore actions, which can be performed by the avatar, for the remote userto select. Upon the remote user selecting a particular action, theremote electronic device 340 transmits the instructions 334 to manuallycontrol the virtual object to the electronic device 310 to perform theselected action.

In certain embodiments, the remote electronic device 340 can generate anindication 348. The indication 348 can be generated in response to theremote electronic device 340 receiving an input from the remote user.The indication 348 notifies the electronic device 310 to modify thevirtual object. For example, the indication 348 instructs the electronicdevice 310 to modify the virtual object if the virtual object and aphysical object at the location of the viewer overlap or collide. Theremote electronic device 340 transmits the information 332 including thegenerated indication 348 to the electronic device 310.

The collision detection engine 318 of the electronic device 310 detectsa collision between the virtual object and a physical object. Forexample, based on the location of the virtual object at the location ofthe viewer and the physical object at the location of the viewer, thecollision detection engine 318 identifies whether the two objectsoverlap. When the two objects (the virtual object and the physicalobjects) overlap, such as when the virtual object moves to overlap orcollide with a physical object (or is within a predefined thresholddistance), and the indication included in the information 332 isreceived, the virtual object adjustment engine 319 modifies the virtualobject.

In certain embodiments, the modification includes moving the virtualobject to a new location within the AR environment that does not overlapa physical object. In certain embodiments, the modification includesaltering the pose of the virtual object to interact with the physicalobject. For example, if the physical object is identified as a wall andthe virtual object is an avatar, the virtual object adjustment engine319 can modify the pose of the avatar to lean against the wall. Inanother example, if the physical object is identified as a chair and thevirtual object is an avatar, the virtual object adjustment engine 319can modify the pose of the avatar to sit in the chair.

In certain embodiments, the modification includes providing overlaysover the real object for indicating a realistic interaction. The overlapcan include a shadow. For example, if the avatar leans against a wall,the shadow can be generated along the wall. In another example, if theavatar sits in a chair, the overlay can include shadows on the seat ofthe chair or the pillows. The overlay also can modify the appearance ofthe physical object. For example, if the avatar sits in a plush sofa,the virtual object adjustment engine 319 can generate an overlayrepresenting that the avatar descends slightly into the sofa.

In certain embodiments, the modification includes generating anappropriate audio effect based on the interaction. For example, if theavatar leans against a wall, a “thud” sound can be generated based onthe properties of the wall specifying that the wall is hard. In anotherexample, if the avatar sits in a plush sofa, a different sound can begenerated indicating that the descends slightly into the sofa.

Although FIG. 3 illustrates one example of the communication system 300,various changes may be made to FIG. 3 . For example, the communicationsystem 300 can receive various types of virtual objects, identifyvarious types of physical objects, and alter a pose or position of avirtual object.

FIG. 4 illustrates example properties associated with a physical objectin accordance with this disclosure. The embodiments shown in FIG. 4 arefor illustration only. Other embodiments can be used without departingfrom the scope of the present disclosure.

As illustrated, FIG. 4 describes four types of properties including: ahard property 410 representing hardness; an elastic property 420representing elasticity; a density property 430 representing density;and an absorption property 440 representing force absorption. The fourproperties are illustrated by an object 402 colliding with a surface,such as surface 412, 414, 422, 424, 432, 434, 442, and 444, of aphysical object with a particular value of one of the properties.

The hard property 410 describe hardness of a physical object. Hardnessis a resistance to deformation. A low hardness value 410 a indicatesthat the object is not resistant to deformation under the weight andforce of other objects. In contrast, a high hardness value 410 bindicates that the object surface is resistant to deformation under theweight and force of other objects. The hard property 410 describes theweight limit (or magnitude of applied force) before deformation isperceived on the surface of an object. For example, a surface 412 willnot show signs of deformation until a weight or force greater than acertain hardness value collides with the surface 412. For instance, thesurface of a physical object will not deform until an object with agreater force or weight is applied thereto. In certain embodiments, thevirtual object adjustment engine 319 of FIG. 3 can modify the appearanceof a physical object by generating and applying an overlay to theobject. That is, the virtual object adjustment engine 319 will notgenerate and apply the overlay, for virtually deforming the physicalobject, until the virtual object places a virtual weight or virtualforce that is greater than certain hardness values, such as threshold orspecified, as indicated by the real-world object colliders.

The elastic property 420 describes elasticity of a physical object.Elasticity is the deformability of an objects surface. A valueassociated with the elastic property 440 defines how much an objecttends to deform or curve when the object is being deformed based on theweight or force of another object. For example, a low elasticity value420 a indicates that the surface of the object deforms less than asurface with a high elasticity value 420 b. The elastic property 420modifies the position of the colliding object. In certain embodiments,the virtual object adjustment engine 319 of FIG. 3 can modify theappearance of a virtual object by sinking the virtual object into thephysical object based on an elasticity value of the physical object. Forexample, when the physical object has a high elasticity value 420 b, thevirtual object adjustment engine 319 modifies the appearance of thevirtual object by sinking the virtual object into the physical object.Alternatively, when the physical object has a low elasticity value 420a, the virtual object adjustment engine 319 does not modify theappearance of the virtual object and places the virtual object on top ofthe physical object.

The density property 430 describes the density of a physical object.Density describes whether the physical object is hollow such that aneffect is caused for sound generation of an interaction between thevirtual object and the physical object. Based on a force that thevirtual object interacts with the physical object, and the density ofthe physical object, certain sounds can be accessed in a sound libraryin the information repository 320 of FIG. 3 . For example, a physicalobject with a high density value 430 b corresponds to hollow sounds 438while a low density value 430 a corresponds to sharper sounds 436. Thesound library can include a spectrum of sounds pertaining to differentdensities of objects as well as objects of different materials. Theforce amplitude and the length of the collision between the virtualobject and the physical object can correlate to the volume and length ofthe sound that is produced via the speaker 290.

The absorption property 440 describes force absorption. Force absorptionindicates how much the physical object can absorb a force that isapplied to the physical object. For example, if a ball bounces on theobject, the value associated with property 440 describes how the ballwill react after the collision. For example, a low force absorptionvalue 440 a indicates that the surface of the object does not absorb theforce, such that the velocity of the ball is not dampened due to thecollision. In contrast, a high force absorption value 440 b indicatesthat the surface of the object absorbs the force, such that the velocityof the ball is reduced. In certain embodiments, the virtual objectadjustment engine 319 of FIG. 3 can modify the how a virtual objectreacts when colliding with a physical object. For example, if thephysical object has a low force absorption value 440 a, then the virtualobject adjustment engine 319 can apply an opposite force, equal inamplitude of the colliding force to the virtual object, as the virtualobject bounces off of the physical object. In contrast, if the physicalobject has a high force absorption value 440 b, then the virtual objectadjustment engine 319 can apply an opposite force that is less than thecolliding force between the virtual object and the physical object, asthe virtual object bounces off of the physical object.

Although FIG. 4 illustrates four example properties associated with anobject, various changes may be made to FIG. 4 . For example, additionalproperties can be associated with a physical object.

FIG. 5 illustrates an example block diagram 500 for adjusting a pose ofan avatar in accordance with this disclosure. The block diagram 500 canbe implemented any of the client devices 106-116 or the server 104 ofFIG. 1 , the electronic device 201 or the server 206 of FIG. 2 , or anycombination thereof. In certain embodiments, the block diagram 500 isperformed by the virtual object adjustment engine 319 of FIG. 3 . Inother embodiments, the block diagram 500 is performed by a remoteelectronic device such as the remote electronic device 340.

The electronic device 310 of FIG. 3 compares the location of the virtualobject to the location of the identified physical objects in the area.When the virtual object is within a threshold distance from theidentified physical objects, and the indication included in theinformation 332 is received (indicating that the remote user allows theelectronic device 310 to modify the virtual object), the electronicdevice 310 modifies the virtual object to interact with the object.Alternatively, when the virtual object is within a threshold distancefrom the identified physical objects, and the electronic device 310receives instructions 334 for manually controlling the virtual object,the electronic device 310 modifies the virtual object to interact withthe object.

In certain embodiments, the virtual object interacts with a physicalobject by modifying a pose of the virtual object, modifying theperception of the physical object, generating a sound corresponding tothe interaction, or a combination thereof. Instead of the virtual objectintersecting with and overlapping the real world object, the pose of thevirtual object is modified to interact realistically with the objectaccordingly to the object type. For example, to modify the pose of thevirtual object, the virtual object can lean against the physical object,sit on the physical object (such as when the physical object is forsitting, such as a chair), climb on the physical object (such as whenthe physical object is for climbing, such as a step stool), and thelike.

As illustrated in the block diagram 500 of FIG. 5 , the electronicdevice 310 performs a pose query 510 to prior to adjusting the pose ofthe virtual object for interaction with the physical object. That is,when the virtual object is within a threshold distance from the physicalobject (such as when the virtual object is near or overlapping thephysical object), the pose query 510 identifies collision position 512.The collision position 512 is the position of the physical object withrespect to the virtual object when the two objects are within athreshold. The pose query 510 also identifies the virtual objectposition 514. For example, if the virtual object is an avatar, thevirtual object position 514 indicates whether the avatar is standing,sitting, laying down, running, jumping, and the like. If the virtualobject is a ball, the virtual object position 514 indicates whether theball is being thrown, rolling, stationary, bouncing, and the like. Thepose query 510 further identifies the physical object type via theobject recognition such as the object recognition engine 314 of FIG. 3 .

Based on the position of the virtual object with respect to the physicalobject (via the collision position 512), the virtual object position514, and the object type 516, the electronic device 310 parses through apose database 520 to select modified pose information. The modified poseinformation includes pose data for altering the pose of the virtualobject, a pose position, a pose mask, and the like. For example, if thevirtual object is an avatar, that is standing in the middle of a sofa atthe viewers location, the electronic device 310 can select modified poseinformation 522 that modifies the pose of the avatar from a standingposition to a seated position on a cushion of the sofa. For anotherexample, if the virtual object is an avatar that is near a wall, theelectronic device 310 can select modified pose information 522 thatmodifies the hands of the avatar to grab or push against the wall. Themodified pose information is provided to the inverse kinematic (IK)solver 534, the virtual object position solver 536, and the overlayadjustment solver 538.

The IK solver 534 obtains the modified pose information 522 and theobject colliders 532. The IK solver 534 modifies the joints of thevirtual object to perform the modified pose. For example, if the virtualobject is an avatar that is standing and the modified pose informationindicates that the avatar is to sit, the IK solver 534 modifies the hipand knee joints of the avatar to transition the avatar from the standingposition to the seated position.

The virtual object position solver 536 obtains the virtual objectposition 514 and the object property data 530, such as via the physicalproperties information 324 of FIG. 3 . The virtual object positionsolver 536 determines how to modify the position of the virtual objectbased on the property data of the physical object.

The overlay adjustment solver 538, obtains the object type 516 and theobject property data 530 and generates a virtual overlay on the physicalobject. For example, the virtual overlay can include a shadow such aswhen the virtual object touches the physical object. For anotherexample, the virtual overlay can indicate that the virtual objectdepresses (sinks) into the object, such as when the properties of theobject indicate that the object has a low hardness and a highelasticity. Alternatively, if the properties of the object indicate thatthe object possesses a high hardness and a low elasticity then thevirtual overlay can indicate a shadow such as when the virtual object isplaced on top of the physical object. The virtual object when displayedto a viewer appears to realistically interact with the physical object.

Although FIG. 5 illustrates one example block diagram, various changesmay be made to FIG. 5 . For example, the block diagram 500 could beperformed by the electronic device 310 of FIG. 3 , the remote electronicdevice 340 of FIG. 3 , or a server such as the server 104 of FIG. 1 .

FIGS. 6A-7G illustrate example images of object interaction between avirtual avatar and a physical object in accordance with this disclosure.In particular, the examples shown in FIGS. 6A-6F illustrate a virtualobject, interacting with a wall that is located at the viewers location,while the examples shown in FIGS. 7A-7G illustrate a virtual objectinteracting with a piece of furniture such as a sofa that is located atthe viewers location.

FIG. 6A illustrates a view of an area 600 at a location at theelectronic device 310. The area 600 includes a wall 602 and a door 604.FIG. 6B illustrates primitives 606 a and 606 b highlighting a portion ofthe wall 602. For example, the object recognition engine 314 of FIG. 3identifies the object as a wall and identifies properties associatedwith it. FIG. 6C illustrates the virtual object, illustrated as anavatar 608 intersecting the wall 602. The virtual object adjustmentengine 319 of FIG. 3 can adjust the position and pose of the avatar 608.For example, as illustrated in FIG. 6D, the avatar 608 is moved to theside of the wall 602, so it is not overlapping the wall 602.Additionally, the pose of the avatar 608 is adjusted to show the avatar608 leaning against the wall 602, as illustrated in FIG. 6D. FIG. 6Eillustrates example overlays 610 a and 610 b. In the examples shown, theoverlays 610 a and 610 b are shadows that are virtually generated anddisplayed on the wall 602. FIG. 6F graphically illustrates a sound 612that is generated and played via the speakers when the avatar 608 placesits virtual hand against the wall 602.

FIG. 7A illustrates a view of an area 700 at the location at theelectronic device 310. The area 700 includes a sofa 702 and two pillows704 a and 704 b. FIG. 7B illustrates primitives 706 a, 706 b, 708 a, and708 b highlighting a portion of the sofa 702 and the two pillows 704 aand 704 b. For example, the object recognition engine 314 of FIG. 3identifies the objects as a sofa and pillows. The object recognitionengine 314 also identifies various properties associated with the sofa702 and pillows 704 a and 704 b, such as hardness, elasticity, density,and the like. FIG. 7C illustrates a view of the remote user 710 atremote location while FIG. 7D illustrates a view of the area 700 of thevirtual object representing the remote user, illustrated as an avatar714. In particular, FIG. 7C illustrates an example view of the remoteuser 710 with an HMD standing within the collision primitive 712representing the sofa 702 where the avatar 714 representing the remoteruser 710 is positioned. FIG. 7D illustrates a point of view of theviewer viewing the avatar 714 of the remote user 710 standing in thesofa 702 at the location of the viewer.

FIGS. 7E, 7F, and 7G illustrate various modifications representing aninteraction between the virtual object and the object based on theproperties of the object. The pose of the avatar 714 is adjusted to showthe avatar sitting on the sofa 702, at illustrated in FIG. 7E. FIG. 7Fillustrates example overlays 716. The overlay 716 includes shadows thatare virtually generated and displayed on the pillows 704 b. The avatar714 can be displayed to depress slightly into the sofa 702, the pillows704 a and 704 b, or both, based on the properties associated therewith.FIG. 7G illustrates sound 718 that is generated and played via thespeakers when the avatar 714 sits in on the sofa 702. The sound 718 canbe based on the identified properties of the sofa. As illustrated, thelines representing the sound 718 are fuzzy and wide representing onetype of sound while the lines representing the sound 612 of FIG. 6F arethin and sharp representing another type of sound.

FIG. 8 illustrates an example method 800 for augmented reality objectinteraction in accordance with this disclosure. The method 800 may beperformed by the server 104 or any of the client devices 106-114 of FIG.1 , the electronic device 201 or the server 206 of FIG. 2 , or any othersuitable device or system. For ease of explanation, the method 800 canbe performed by the electronic device 310 of FIG. 3 . However, themethod 800 can be used with any other suitable system or device.

In block 802, the electronic device 310 identifies an object within anarea. For example, the electronic device 310, captures images or videosof the area or obtains the images or videos. The electronic device 310then performs object recognition for identifying a category or type ofobject within the area. The electronic device 310 also identifiesproperties associate with the identified object. The properties caninclude hardness, elasticity, density, force absorption, and the like.Hardness indications the level of resistance to deformation when anotherobject collides with identified object. Elasticity indicates a level ofdeformation of the identified object when another object collides withidentified object. Density indicates a type of sound that is associatedwith the identified object when another object collides with identifiedobject. Force absorption indicates a level of force that the identifiedobject absorbs when another object collides with identified object.

In certain embodiments, the electronic device 301 generates objectinformation to be transmitted to a remote electronic device. The objectinformation includes the identified object and properties associatedwith the object. The object information also includes an indication of alocation of the object within the area. The object information canfurther include primitives around an identified object. The primitivescan provide to a remote user a shape of the object.

In block 804, the electronic device 310 transmits the object informationto the remote electronic device. The remote electronic device can takethe object information and generate a virtual representation of theobject at the location of the electronic device 310. In block 806, theelectronic device 301 receives from the remote electronic deviceinformation. The information can include a virtual object, such as anavatar that represents the remote user. The electronic device 301 canalso receive an indication as to whether the electronic device 301 is tomodify the virtual object if the virtual object collides or overlapswith a physical object (such as the identified object) at the locationviewer. In block 808, the electronic device 310 displays the receivedvirtual object.

In certain embodiments, the virtual object is controlled by a remoteuser via the remote electronic device. For example, the remote user canmove the virtual object within the area of the electronic device 310.The movements of the virtual object can be displayed in real time on adisplay of the electronic device 310 for the user to view.

In block 810, the electronic device 310 determines that when the virtualobject is within a threshold distance from the identified object, thevirtual object is modified to interact with the identified object basedon the properties associated with the virtual object. For example, ifthe virtual object is an avatar, the pose of the avatar can be adjusted.For instance, if the object is identified as a chair, the avatar can beadjusted to sit in a chair. If the object is identified as a wall, theavatar can be adjusted to lean against the wall.

In certain embodiments, an overlay is generated and displayed based onthe interaction between the virtual object and the identified object.For example, a shadow can be displayed over a part of object based onthe location of the virtual object. In another example, an overlay candepict the virtual object sitting on a sofa that depresses slightlybased on the virtual object sitting on a plush sofa.

In certain embodiments, a sound is generated based on the interactionbetween the virtual object and the identified object. The sound isselected based on a property of the object. For example, if the objectis metal and dense, a first sound is generated, and if the object ismetal and hollow another sound is made.

The above flowcharts illustrate example methods that can be implementedin accordance with the principles of the present disclosure and variouschanges could be made to the methods illustrated in the flowchartsherein. For example, while shown as a series of steps, various steps ineach figure could overlap, occur in parallel, occur in a differentorder, or occur multiple times. In another example, steps may be omittedor replaced by other steps.

Although the figures illustrate different examples of user equipment,various changes may be made to the figures. For example, the userequipment can include any number of each component in any suitablearrangement. In general, the figures do not limit the scope of thisdisclosure to any particular configuration(s). Moreover, while figuresillustrate operational environments in which various user equipmentfeatures disclosed in this document can be used, these features can beused in any other suitable system.

None of the description in this application should be read as implyingthat any particular element, step, or function is an essential elementthat must be included in the claim scope. The scope of patented subjectmatter is defined only by the claims. Moreover, none of the claims isintended to invoke 35 U.S.C. § 112(f) unless the exact words “means for”are followed by a participle. Use of any other term, including withoutlimitation “mechanism,” “module,” “device,” “unit,” “component,”“element,” “member,” “apparatus,” “machine,” “system,” “processor,” or“controller,” within a claim is understood by the applicants to refer tostructures known to those skilled in the relevant art and is notintended to invoke 35 U.S.C. § 112(f).

Although the present disclosure has been described with exemplaryembodiments, various changes and modifications may be suggested to oneskilled in the art. It is intended that the present disclosure encompasssuch changes and modifications as fall within the scope of the appendedclaims. None of the description in this application should be read asimplying that any particular element, step, or function is an essentialelement that must be included in the claims scope. The scope of patentedsubject matter is defined by the claims.

What is claimed is:
 1. A method for augmented reality object interactioncomprising: identifying an object within an area and propertiesassociated with the object; transmitting, to a remote electronic device,object information associated with the object; receiving, from theremote electronic device, information including a virtual object,representing a remote user; displaying the virtual object within thearea; and in response to determining that the virtual object is within athreshold distance from the object, modifying the virtual object tointeract with the object based on the properties of the object.
 2. Themethod of claim 1, wherein the properties include at least one of: ahardness property indicating a level of resistance of the object todeformation when the virtual object interacts with the object; anelasticity property indicating a level of deformation of the object whenthe virtual object interacts with the object; a density propertyindicating a level of sound that is generated when the virtual objectinteracts with the object; or a force absorption property indicating alevel of force that the object absorbs when the virtual object interactswith the object.
 3. The method of claim 1, further comprising:generating the object information associated with the object, the objectinformation including (i) at least one predefined shape outlining aperimeter of the object, (ii) a location of the object within the area,and (iii) the properties associated with the object, wherein the atleast one predefined shape outlining the perimeter of the objectprovides an indication of a shape of the object at the location of theobject in the area for the remote user; receiving, from the remoteelectronic device, an indication to modify the virtual object; andmoving the virtual object within the area based on the information thatis received in real time from the remote electronic device.
 4. Themethod of claim 1, wherein modifying the virtual object to interact withthe object comprises adjusting a pose of the virtual object and aposition of the virtual object based on the properties of the object. 5.The method of claim 1, wherein modifying the virtual object to interactwith the object comprises: generating a sound effect representing theinteraction; and displaying a visual indication representing the soundeffect at a location of the interaction.
 6. The method of claim 1,wherein modifying the virtual object to interact with the objectcomprises: generating an overlay associated with the interaction basedon the properties of the object; and displaying the overlay over theobject.
 7. The method of claim 1, further comprising: generating atleast one predefined shape outlining a perimeter of the object;determining that the virtual object overlaps one of the at least onepredefined shape outlining the perimeter of the object; and identifyinga pose for the virtual object based on a position of the virtual objectwith respect to the object and the properties of the object, whereinmodifying the virtual object to interact with the object, comprises:when the object is identified as a wall, modifying the virtual object tolean against the wall based on the identified pose, and when the objectis identified as a seat, modifying the virtual object to sit in the seatbased on the identified pose.
 8. An electronic device for augmentedreality object interaction comprising: a communication interface; and aprocessor operably connected to the communication interface, theprocessor configured to: identify an object within an area andproperties associated with the object; transmit, to a remote electronicdevice, object information associated with the object; receive, from theremote electronic device, information including a virtual object,representing a remote user; display the virtual object within the area;and in response to determining that the virtual object is within athreshold distance from the object, modify the virtual object tointeract with the object based on the properties of the object.
 9. Theelectronic device of claim 8, wherein the properties include at leastone of: a hardness property indicating a level of resistance of theobject to deformation when the virtual object interacts with the object;an elasticity property indicating a level of deformation of the objectwhen the virtual object interacts with the object; a density propertyindicating a level of sound that is generated when the virtual objectinteracts with the object; or a force absorption property indicating alevel of force that the object absorbs when the virtual object interactswith the object.
 10. The electronic device of claim 8, wherein theprocessor is further configured to: generate the object informationassociated with the object, the object information including (i) atleast one predefined shape outlining a perimeter of the object, (ii) alocation of the object within the area, and (iii) the propertiesassociated with the object, wherein the at least one predefined shapeoutlining the perimeter of the object provides an indication of a shapeof the object at the location of the object in the area for the remoteuser; receive, from the remote electronic device, an indication tomodify the virtual object; and move the virtual object within the areabased on the information that is received in real time from the remoteelectronic device.
 11. The electronic device of claim 8, wherein tomodify the virtual object to interact with the object, the processor isconfigured to adjust a pose of the virtual object and a position of thevirtual object based on the properties of the object.
 12. The electronicdevice of claim 8, wherein to modify the virtual object to interact withthe object, the processor is configured to: generate a sound effectrepresenting the interaction; and display a visual indicationrepresenting the sound effect at a location of the interaction.
 13. Theelectronic device of claim 8, wherein to modify the virtual object tointeract with the object, the processor is configured to: generate anoverlay associated with the interaction based on the properties of theobject; and display the overlay over the object.
 14. The electronicdevice of claim 8, wherein: the processor is further configured to:generate at least one predefined shape outlining a perimeter of theobject, determine that the virtual object overlaps one of the at leastone predefined shape outlining the perimeter of the object, and identifya pose for the virtual object based on a position of the virtual objectwith respect to the object and the properties of the object; and tomodify the virtual object to interact with the object, the processor isconfigured to: when the object is identified as a wall, modify thevirtual object to lean against the wall based on the identified pose,and when the object is identified as a seat, modify the virtual objectto sit in the seat based on the identified pose.
 15. A non-transitorymachine-readable medium containing instructions that when executed causeat least one processor of an electronic device to: identify an objectwithin an area and properties associated with the object; transmit, to aremote electronic device, object information associated with the object;receive, from the remote electronic device, information including avirtual object, representing a remote user; display the virtual objectwithin the area; and in response to determining that the virtual objectis within a threshold distance from the object, modify the virtualobject to interact with the object based on the properties of theobject.
 16. The non-transitory machine-readable medium of claim 15,wherein the properties include at least one of: a hardness propertyindicating a level of resistance of the object to deformation when thevirtual object interacts with the object; an elasticity propertyindicating a level of deformation of the object when the virtual objectinteracts with the object; a density property indicating a level ofsound that is generated when the virtual object interacts with theobject; or a force absorption property indicating a level of force thatthe object absorbs when the virtual object interacts with the object.17. The non-transitory machine-readable medium of claim 15, furthercontaining instructions that when executed cause the at least oneprocessor to: generate the object information associated with theobject, the object information including (i) at least one predefinedshape outlining a perimeter of the object, (ii) a location of the objectwithin the area, and (iii) the properties associated with the object,wherein the at least one predefined shape outlining the perimeter of theobject provides an indication of a shape of the object at the locationof the object in the area for the remote user; receive, from the remoteelectronic device, an indication to modify the virtual object; and movethe virtual object within the area based on the information that isreceived in real time from the remote electronic device.
 18. Thenon-transitory machine-readable medium of claim 15, wherein theinstructions that when executed cause the at least one processor tomodify the virtual object to interact with the object compriseinstructions that when executed cause the at least one processor toadjusting a pose of the virtual object and a position of the virtualobject based on the properties of the object.
 19. The non-transitorymachine-readable medium of claim 15, wherein the instructions that whenexecuted cause the at least one processor to modify the virtual objectto interact with the object comprise instructions that when executedcause the at least one processor to: generate a sound effectrepresenting the interaction; and display a visual indicationrepresenting the sound effect at a location of the interaction.
 20. Thenon-transitory machine-readable medium of claim 15, wherein theinstructions that when executed cause the at least one processor tomodify the virtual object to interact with the object comprisesinstructions that when executed cause the at least one processor to:generate an overlay associated with the interaction based on theproperties of the object; and display the overlay over the object.